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DECLARATION UNDER 37 C.F.R. § 1.131 



U.S. Patent and Trademark Office 
Commissioner for Patents 



We, Kenneth P. Hinckley, Steven N. Bathiche and William Vong, hereby declare as 
follows: 

1) We are named as joint inventors of the above- identified patent application. 

2) We are presently employed by Microsoft Corporation (Microsoft), and were 
employed by Microsoft during conception and development of the inventions in the 
above- identified application, 

3) Microsoft is the assignee of the above-identified application, 

4) Prior to October 6, 2000, the filing date of U.S. Patent No. 6,680,677, wc actually 
reduced to practice in the United States the invention recited in pending claims 1-31 
of the above- identified application. 

5) Exhibit A shows two slides from a PowerPoint presentation entitled "Touch- 
Sensing Input Devices" (dates redacted) along with the cover slide that Ken 
Hinckley along with Mike Sinclair presented, which introduced the core ideas we 
conceived and invented In slides 23 and 45, there is a reference to tooltips 
appearing when a button is touched and the notion of a time delay before the tooltip 
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fades and the applicability of the invention to a keyboard and a mouse as input 
devices. Also, a representative tooltip which indicates the functionality of 
"Launches My Computer" is shown in slide 23. Exhibit B, Touch 1 Usability 
Protocol (dates redacted), describes the process that Tim Muss, who was a usability 
engineer designated to test our prototype, employed for a usability test that was to 
be conducted with 12 subjects on a touch sensitive mouse according to our 
invention. Exhibit C, the Usability Report (dates redacted), was generated based on 
the actual usability test that was conducted as outlined in Exhibit B. Portions of 
Exhibit B provide background details including the Mouse Button tooltips portion 
of Section 2.0 and Section 4.0, Touch spotlight. 

6) Exhibit C when read in view of the referenced portions of Exhibit A as well as 
Exhibit establishes that we had reduced to practice the invention of independent 
claim 1. More specifically, as referred to in the second paragraph on page 4 of 
Exhibit C four features of the touch mouse were investigated including touch 
tooltips where a user placed a hand close to or on a mouse; "Touch of 
programmable mouse buttons (4 and 5) provided an application- sensitive tooltip 
identifying the action of the button for the current application." This section of 
Exhibit C provides support for each of the claim 1 steps including detecting a 
presence proximate to or contacting a first auxiliary control for a predefined period 
in which the first auxiliary control maintains an inactive state and generating 
feedback responsive to the detecting, the feedback providing an indication of the 
functionality of the first auxiliary control, the functionality of the first auxiliary 
control and associated feedback being dependent upon which of a plurality of first 
application programs is active. Further support for claim 1 can be found in Exhibit 
C at, among other places, pages 10-1 1, where participant reaction to touch tooltips 
is discussed. For example, inadvertent actuation of buttons is discussed. 

7) Independent claim 9 calls for detecting a first physical presence proximate to or 
contacting the auxiliary control for a predefined period in which the auxiliary 
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control maintains an inactive state. Reduction to practice of this feature is described 
in the above paragraph with respect to Exhibit C. Claim 9 also recites the step of 
generating feedback responsive to the step of detecting, the feedback providing an 
indication of the functionality of the auxiliary control, the generating further 
including displaying a display widget on the display screen responsive to the step of 
detecting, wherein the display widget identifies a text macro, which is a block of 
text assigned to the first auxiliary control, and displays at least a portion of text 
corresponding to the text macro. Exhibit C as discussed in the above paragraph 
evidences reduction to practice of the claim 9 step of generating feedback provid ing 
an indication of the functionality of an auxiliary control including displaying a 
display widget. A tool tip is an illustrative display widget. Although Exhibit C does 
not evidence that the display widget identifies a text macro, which is a block of text 
assigned to the first auxiliary control, and displays at least a portion of text 
corresponding to the text macro as called for in claim 9, Exhibit D entitled "Patent 
Disclosure" (dates redacted) on page 4, third full paragraph shows and describes a 
text macro. Namely, Exhibit D describes that a text macro could be created where a 
block of text would be inserted when a hotkey is depressed. More pertinently, 
Exhibit D goes on to describe that when a key is assigned a text macro, by touching 
a key a user can be presented with an on screen display indicating what function (in 
this instance text macro) is assigned to the key and the actual text macro An 
illustrative on screen display is depicted between the third and fourth full 
paragraphs on page 4 of Exhibit D. We submit that it wou ld have been apparent to 
those skilled in the art that the reduction to practice discussed in the above 
paragraph regarding claim 1 with respect to Exhibit C could have been easily 
modified and/or it would have been obvious as to how to modify our touch sensitive 
mouse to achieve and realize displaying the at least a portion of text corresponding 
to a text macro when the text macro was the assigned function of the key 
functionality as we claimed and as we conceived and invented as evidenced by 
Exhibit C 
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8) Independent claim 18 includes detecting a first physical presence proximate to or 
contacting a first auxiliary control for a predefined period in which the first 
auxiliary control maintains an inactive state and generating first feedback 
responsive to the step of detecting, the first feedback providing an indication of the 
functionality of the first auxiliary control. These features have been reduced to 
practice as discussed above with respect to claim 1 and Exhibit C Claim 18 further 
calls for detecting a second physical presence proximate to or contacting a second 
auxiliary control different from the first auxiliary control in which the second 
auxiliary control maintains an inactive state while detecting the first physical 
presence proximate to or contacting the first auxiliary control, and generating 
second feedback responsive to the step of detecting the second physical presence, 
the second feedback indicating functionality associated with the combination of the 
first auxiliary control and the second auxiliary control. Exhibit D on page 3, in the 
second full paragraph describes the steps of detecting a second physical presence 
and generating second feedback responsive to detecting the second physical 
presence, with the second feedback indicating functionality of the combination of 
the first and second auxiliary controls, the so-called new control. We submit that it 
would have been apparent to those skilled in the art that the reduction to practice 
discussed regarding claim 1 in the above paragraph with respect to Exhibit C could 
have been easily modified and/or it would have been obvious as to how to modify 
our touch sensitive mouse to achieve and realize the claim 1 8 steps of detecting the 
second physical presence while detecting the first physical presence and generating 
the second feedback indicating functionality associated with the combination of the 
first auxiliary control and the second auxiliary control as we claimed and as we 
conceived and invented as evidenced by Exhibit C. 

9) Independent claim 28 calls for detecting a first physical presence proximate to or 
contacting a first auxiliary control without activating the first auxiliary control; and 
displaying a first display widget on the display screen responsive to the step of 
detecting, the first display widget providing a tool tip associated with the first 
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auxiliary control, wherein the tool tip is a textual label. Support for these features is 
at least coextensive with the support identified for the features of claim 1 as 
discussed above and includes the portions of Exhibit A providing background 
details include the Mouse Button tooltips portion of Section 2.0 and Section 4.0, 
Touch spotlight, and Exhibit C on pages 4, 10 and 1 1 Claim 20 further calls for the 
tool tip to indicate one of an identity of a user, tuning of an audio application, 
tuning of a video application, volume control, control of a feature with multiple 
settings, a control function corresponding to a key combination, and an application 
that will be launched by activating the first auxiliary control. As discussed above, 
Exhibit D on page 3 in the second full paragraph describes a tool tip indicating a 
control function corresponding to a key combination. We submit that it would have 
been apparent to those skilled in the art that the reduction to practice discussed 
regarding claim 1 in the above paragraph with respect to Exhibit C could have been 
easily modified and/or it would have been obvious as to how to modify our touch 
sensitive mouse to achieve and realize a tool tip indicating a control function 
corresponding to a button (key) combination as we conceived and invented as 
evidenced by Exhibit C. 

10) As discussed above in connection with Exhibits A, B, C and D, the features of 
independent claims 1,9, 18 and 20 were reduced to practice or it would have been 
apparent to those skilled in the art as to how the claimed features reduced to 
practice could have been easily modified and/or it would have been obvious as to 
how to modify our device to achieve and realize the other claimed features. Also, 
many of the dependent claim features were also reduced to practice as evidenced by 
Exhibits A, B, C and D. To the extent that we may not have reduced to practice 
every dependent claim feature, we submit that it would have been apparent to those 
skilled in the art that we had established a working device which could have easily 
been modified to realize any such feature and/or that it would have been obvious as 
to how to modify our device to achieve and realize such a feature. 
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11) Exhibits A, B, C and D have not been altered since they were originally prepared 
except for the redaction of references to dates. 

12) Each of us individually represents that we are over 18 years of age and of 
competent mind 

13) All statements made of our own knowledge are true and all statements made on 
information and belief arc believed to be true; and further, these statements were 
made with the knowledge that willful, false statement so made are punishable by 
fine or imprisonment or both, under 18 U.S C, § 1001 and that such willful, false 
statements may jeopardize the validity of the above-identified application or any 
patent issuing thereon. 

Respectfully submitted, 



/Kenneth P. Hinckley/ 



Feb. 5, 2007 



Kenneth P Hinckley 



Date 



Steven N. Bathiche 



Date 



William Vong 



Date 
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EXHIBIT B 



Tim Muss 

Hardware Design Group 



Touch 1 Usability Protocol 



Study Dates; | 

Subjects; 12 (6 expert, 6 advanced mouse users) 
1.0 Introduction 

The study focuses on three implementations of touch: fading toolbars in Word, cursor location 
(sonar), and tool tips for default back and forward mouse buttons. 



ZO Research Objectives 
Fading Toolbars 

► User feedback to fading toolbars when hand is removed from or placed on mouse. 

■ Speed of fade in/ out 

■ Does it fade when expected? 

■ Pros/cons 

■ Value of this feature 

Sonar 

► User feedback to hiding cursor when hand is removed from mouse 

► User feedback to using sonar-like display to help locate cursor upon touch of mouse 

■ Can they find cursor easier than without sonar? 

■ Does sonar provide too much/too little assistance? 

■ Pros /cons 

■ Value of this feature 
Mouse Button Tool tips 

► User feedback of touching mouse button for tool tips. 

■ Does the tool tip appear/ fade too fast? 

■ Location of too 1 tip 

■ Inadvertent actuation? 

■ Pros/ Cons 

■ Value of this feature 



Touch Technology Feedback 

► What are their expectations for touch technology? How should it be used? 
What do users think of applying touch to mouse or keyboard? 



Tim Muss 

Hardware Design Group 



1.0 Opening Interview 

General 

What mouse(s) do you use most often? 

Would you say you use your keyboard or your mouse more often? 
What software applications to you use most often? 
What do you dislike most about your current mouse? 
What do you think of the mouse-over tool tips? 

Physical Mouse Use 

Do you rest your hand on the mouse when reading/ thinking? 
How do you hold the mouse? What part of the hand touches the mouse? 
Where do you rest your fingers/ hands when using your mouse? On buttons? 
Mouse Use with Software 

In general, list some things you use your mouse for? (i.e. navigation, wake up computer, selection, 

etc) 

Does the mouse cursor ever get in the way when you are entering text? 

Have you ever had trouble finding your mouse cursor on the screen? 

How often do you use the toolbars? Name the most common features you use. 

Have you ever programmed the buttons on your mouse to perform different functions than the 
default settings? 



Tim Muss 
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2.1? fradittg r<x»lb*r» 

1 Equipment: System ; . K i .\ 1 ^co-. ■: : a, v\ Jk W o <d 
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Objective measure 

► Time to complete task for with and without touch 

Subjective measures 

1. What are your initial reactions to the la-ling toolbars? 

2. Did you have trouble urxieistandjrtg that touching the mouse caused the toolbars to fade in 

and out? 

3. Why do you think the upper portion of the screen was gray? 

4. What are the advantages of having fading toolbars? 

5. What are the disadvantages of laving fading toolbars? 

6. Do you think you would use this feature? Why? 

7. Do you prefer the fading toolbars and the to uch mouse to the standard toolbar and mouse? 
Why? 

& Did the toolbars fade in and out when you expected them to? Why? 

9. Would you rate the speed of the fade too last, just right, or too slow? Why? 

10. Did this feature distract you from the task? If yes, explain, 

11, 1 performed the task ...... 

better with the touch mouse O O O O O better without the touch mouse 

12. Based on this feature, I feel the mouse is.,.. 

More touchable with this OOOOO More touchable without this 

touch feature touch feature 



13. Thte feature is.,, 

annoying OOOOO helpful 
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14. To what degree would you think this feature affects the appeal of a mouse? 
This feature makes a mouse less O O O O O O O This feature makes a mouse more 
appealing appealing 



3.0 'Sonar' - cursor location assistance 

I Equipment: 



System 2: SVdom wilh Louch. i\.?k and louch dm-vr*;. Hxcel 



Method 

Users will perform a predetermined task in Excel for about 5 minutes (entering text, moving text, 
scrolling, etc.) on system 1 without using sonar and system 2 with sonar. Users will be asked to use 
Excel toolbars and not keyboard shortcuts or right mouse button. 

Objective measures 

► Document cursor movement behavior. 

■ How does the user leave the cursor when moving from mouse to keyboard? 

Subjective measures 

1. What are your initial reactions to sonar? 

2. What are yo ur initial reactions to the disappearing cursor? 

3. Was it easy for you to notice that the cursor disappeared when you let go of the mouse? 

4. Was the cursor easier, the same, or harder to find with sonar than without? 

5. What are the pros of the disappearing cursor? 

6. What are the cons of the disappearing cursor? 

7. What are the pros of sonar? 

8. What are the cons of sonar? 

9. Do you feel that sonar offers too little, just right, or too much assistance for finding the 
cursor? 

10. Were you distracted from the task by this touch feature? If yes, explain. 



11. 1 performed the task . . . 
better with a disappearing 
cursor 



better without a disappearing 
cursor 



12. 1 performed the task . 
better with sonar 



O O O O O better without sonar 



13. Based on this feature set, I feel the mouse is.. 
More touchable with this O O O O O 

touch feature 



More touchable without this 
touch feature 
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14. The disappearing cursor feature is... 

annoying O O O O O helpful 



15. The sonar feature is. . . 

annoying OOOOO helpful 

16. To what degree would you think the disappearing cursor feature affects the appeal of a mouse? 
The feature makes a mouse less OOOOOOO This feature makes a mouse mpre 

appealing appealing 

17. To what degree would you think the sonar feature affects the appeal of a mouse? 

The feature makes a mouse less OOOOOOO This feature makes a mouse more 
appealing appealing 



4.0 Touch spotlight 

I Equipment: System 1 ■ :<? i \i Sec,**n«. Vi ..k oul. IK, Kcd 

] System 2: Sedona with touch. Viper with touch, W2k and touch drivers. Word, IE. 

| Excel 

Method 

Users will first perform a predetermined task in Excel, Word, and IE for about 5 minutes (entering 
text, moving text, scrolling, etc.) on system 1 without using touch spotlight and system 2 with touch 
spotlight. Users will be asked to use toolbars and not keyboard shortcuts or right mouse button. 
Users will be presented a written scenario describing the pTe-assignment of actions to the / back / and 
'forward' buttons. 

Objective measures 

■ Time to complete task for each of 3 mice 

■ Inadvertent actuations 

Su ixjective measures 

1. What are your initial reactions to getting tool tips for each button by touch? Please comment 
on each of the two touch mice. 

2. Do you fell about touching a button for a tool tip? 

3. Please comment on the location of the tool tip? 

4. Does the tool tip appear too fast, just right, or too slow? 

5. Does the tool tip disappear when expected? 

6. Were there any times when you did not want touching the button to give a tool tip? 

7. Were there any times when you touched a button by accident? 
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8. What are the pros of this feature? 

9. What are the cons of this feature? 
10. 1 performed the task . : 

better with the touch mouse O O O O O better without the touch mouse 



11. Based on this feature and the touch-Sedona mouse, I feel the mouse is, , -, 
More touchable with this O O O O O More touchable without this 

touch feature to uch feature 



12. Based on this feature and the touch-Viper mouse, I feel the mouse is. . . 
More touchable with this OOOOO More touchable without this 

touch feature touch feature 



13. For the touch-Sedona mouse, this feature is. , 

annoying OOOOO helpful 

14. For the to uch- Viper mouse, this feature is. . . 

annoying OOOOO helpful 

15. To what degree would you think this feature affects the appeal of a mouse? 
Sedona 

This feature makes a mouse less O O O O O O O This feature makes a mouse more 
appealing appealing 
Viper 

This feature makes a mouse less O O O O O O O This feature makes a mouse more 
appealing appealing 

Wrap-up Interview 

1. Based on the features you saw today, how well did they meet your expectations? 
Z Can you think of any applications of touch that would better fit your expectations? 

3. Can you give me examples of how you think touch should be used in computer mouse design? 
Computer keyboard design? 

4. Can you think of any downsides to using touch in mo use or key board designs? 
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DATE: 




TO: 



Steve Bathiche, Scott Plank, David Koan, LaShaun 
Bellamy, Touch POC 



FROM: 



Tim Muss 



RE: 



Final Usability Report 



CC: 



Hardware User Research 



TITLE: USING TOUCH ON MOUSE TO CONTROL TOOLBARS, CURSOR, AND 
TOOLTIPS: PHASE I 



EXECUTIVE SUMMARY 

This study investigated how advanced to expert mouse users reacted to using touch 
of the mouse to control the following four major features: Fading toolbars; 
Disappearing/reappearing cursor; Touch tooltips; Sonar. 

Nearly all participants had a difficult time determining that touch controlled 
these features. Most participants thought mouse movement caused the feature 
effects. Others thought movement of the cursor to a particular portion of the screen, 
the proximity of hand to mouse, the heat of hand, the red light on Sedona, or the 
buttons on the mouse caused the feature effects. Others didn't know 

In general, participants felt that these features, especially touch tooltips, 
made the mouse less touchable and that they would change the way they 
interacted with the mouse. Some mentioned they would hold the mouse lighter, 
be more aware of their finger placement, or be anxious of unwanted actions cause by 
touching the mouse. It was noted that implementing touch, as presented here, to 
the Sedona mouse may cause users to interact with the mouse less, which is 
inconsistent with our recent focus to make Microsoft mice more form-fitting, and 
comfortable to use. user research to determine the accuracy of these findings is 
encouraged. 

A majority of the users agreed that they would like to see these features in 
the next mouse they purchase but a clear ranking of features was not 
established. The greatest number of participants indicated they would like to see 
the touch tooltips feature in their next mouse and that the touch tooltips feature 
made the mouse most appealing, was the most helpful, and best fit their work style. 
But, participants indicated that the touch tooltips feature made the mouse the least 
touchable of all features. This may have been a result of participants resting their 
fingers on the touch-sensitive buttons. The study was unable to determine how much 
more, if any, users would pay for these features. 
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Half of the participants felt that the fading toolbars and sonar features were 
more annoying than helpful. Participants were split on whether they would use 
these features or not. Participants appreciated they could read text underneath the 
toolbars (when faded), but clearly reported they did not like that when they touched 
the mouse to edit this text, the toolbars appeared and prohibited them from editing 
it without scrolling. The fading toolbars on scroll was better received than by touch. 
Most participants appreciated the idea of a sonar feature (cursor location assistance), 
but found the present execution of sonar less than optimal. Participants reported that 
these features were, or may become, distracting with prolonged use. 

Reactions to a disappearing/reappearing cursor without sonar were mostly 
positive. The present implementation of this feature was nearly transparent to 
participants. No participant reported having trouble using this feature. Only two felt 
that this feature was more annoying than helpful. However, of the four features 
presented, participants felt this feature contributed least to the appeal of the mouse. 

Participants indicated that they might only use these features for a short 
period. Participants mentioned they wanted more control over feature performance, 
including to turn them off. Participants felt once they learned the programmable 
button configuration, they would want to turn off the touch tooltips. Participants also 
felt that fading toolbars and sonar to be either too distracting or not appropriate to 
have on at all times and would want to turn them off as well. Furthermore, 
participants mentioned they would not use these features if they noticed any 
decrease in system performance. 

Participants felt touch has potential to enhance user experience. It was 

evident touch "wowed' participants and most were excited of the novelty of the 
features after the touch features were explained. One participant mentioned, 'touch 
created a deeper relationship with the mouse' and another, x l like that the mouse is 
responding to me\ This study suggested that participants found touch to be more of 
an emotional connection to the computer than a physical 'switch'. Three participants 
mentioned touch would be a great addition to the gaming experience. Most were 
more excited about the potential of touch than of the features presented in this 
study. 

This study suggests that using touch sensitivity on a mouse to control these 
features may not be the best implementation of touch technology for the 
end-user. Since touch was not easily understandable to users as presented in this 
study, it may be a clear sign this implementation did not match the expectations of 
an office user. Next steps may be to take a broader approach and perform more 
generative research of user expectations, perceptions, and perceived value of touch 
in a variety of contexts to determine which, if any, input device best supports the 
technology. 
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4.0 How do users rank these four features? 7 
5.0 Which feature is most appealing and which feature is least appealing? 7 
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T. Muss 

Using Touch on Mouse: Phase I 



BACKGROUND 



This study was conducted on ^^^^ ^^^^^^ ^^^^J- The study follows previous 
work, conducted by Microsoft Research on applying touch sensitivity to input devices 
(Hinckley and Sinclair, 1999: Internal http.//msrweb/groups/ui/Papers/touch- 

sensing.doc) 

The present study investigated how users reacted to using touch of the mouse to 
control four major features. Characteristics of the user pool are described in 
Appendix A. Placing the hand close (<l/2") to or on the mouse (hereafter referred to 
as touch) and removing the hand (hereafter referred to as release) resulted in the 
following actions. 

Fading toolbars Toolbars faded in on touch and faded out on release. 

Scrolling with the mouse scroll wheel or using the side scrollbar caused 
toolbars to fade out. Movement of the mouse caused the toolbars to fade in. 
Disappearing cursor. Cursor disappeared on release and reappeared on 
touch. 

• Sonar. A series of circles zeroed in on the cursor on touch. No action on 
release. 

. Touch tooltips. Touch of programmable mouse buttons (buttons 4 and 5) 
provided an application-sensitive tooltip identifying the action of the button 
for current application Tooltip disappeared on release or after a short 
timeout. 

Research Questions 

The study focused on the following research questions: 

How do users presently use tooltips, toolbars, and their mouse cursor? 
. What is the discoverability of a touch sensitive mouse? Do users understand 

that touching the mouse controlled the above on-screen features? 
. Would users want to see these features in their next mouse purchase? 

• How do users rank these four features? 

. Which feature is most appealing, and which feature is least appealing? 

• Do users find these features helpful or annoying? Do these features fit in with 
their preferred work style? Is there too much going on in the screen? 

• Do users find the mouse more or less touchable? 

. What are the user reactions to each of the above features? 

Methods 

Users were first exposed to one of the following conditions followed by the other. 
Order was counter-balanced. All users completed an initial verbal interview and 
completed paper-based surveys after each condition. 

• Sonar and disappearing cursor with touch tooltips. Users were informed 
of the predetermined assignment of the programmable buttons. (IE: 
back/forward; Excel: copy/ paste; Word: undo/redo). A series of verbal 
commands were given requiring them to switch between the three 
applications, enter text, and use the two programmable buttons. Users 
completed the task using first the Sedona mouse design and followed by the 
Viper mouse design (Appendix B). Initial and post-task reactions to the 
features were recorded. 

• Fading toolbars and disappearing cursor. Users were asked to complete a 
simple form in Word requiring them to scroll, use the toolbars, and enter text 
using the Sedona mouse. Initial and post-task reactions to the features were 
recorded. 
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RESULTS 



1.0 How do users presently use tooltips, toolbars, and their 
mouse cursor? 

What are the major likes and dislikes of participants' present mouse? 

Users appreciate that their present mouse is fast, quiet, ard lightweight. At least half 
of the participants used a scroll wheel and all really enjoyed the feature. All 
participants mentioned they leave their hand on the mouse while reading, thinking, 
or waiting for an application to load- One participant wanted the scroll wheel to have 
a Smooth' setting that wouldn't make scrolling so choppy both mechanically and on- 
screen. One participant mentioned he would like the light on the Sedona mouse to 
turn off at night, even when his computer is on. 

How do participants use tooltips? 

Participants do use tooltips, but rarely. They use them mostly when they don't 
understand what an icon represents and when they are learning a program. They 
also use them to get file information and to see the <alt images html tag while 
viewing web pages. At least three participants mentioned that the tooltip presently 
appears too slow and it takes to much work to make them appear (repositioning 
cursor over icon and waiting) or they stay visible for too long. Other participants 
mentioned tooltips are good for novice users and non-critical information, but 
sometimes the information in the tooltip Isn't useful. Participants wanted more 
information on demand {like the ability to go to a web page for more information), 
consistency in all applications, and the ability to adjust content and speed. Only 
three of ten participants reported that the tooltips have ever gotten in the way. 

How do participants use toolbars? 

All participants mentioned they use toolbars. Only two participants reported they use 
hot-keys more than toolbars. Two participants never turn off the toolbars because 
they like the quick reference. Eight participants rarely turn off toolbars and do so to 
have more screen space or to make screen less distracting when writing. Participants 
appreciate they can customize and undock the toolbars. They would like toolbars to 
me more flexible (i.e. customizable skins for more friendly look and feel, re-size 
icons) and be more consistent between applications. 

Do participants have any issues with their mouse cursor? 

Six subjects mentioned the mouse cursor does get in their way after entering text or 
typing. Two participants mentioned specifically that the T cursor ir Word is most 
annoying. Seven participants mentioned they had trouble finding the mouse cursor 
on the screen as a result of the following: screen glare, a touchy pointing device, 
when mouse cursor goes off the screen, is on similar colored backgrounds, or when 
software is loading. Participants mentioned they didn't want the mouse cursor to be 
able to go off the screen, they wanted to have the mouse cursor center itself after a 
period of non-use, and to be able to change the size of the cursor {i.e. pixel by 
pixel). One participant reported he used his mouse cursor to tell him if his system 
has locked-up. 
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2,0 What is the discoverability of a touch sensitive mouse? Do 
users understand that touching the mouse controlled the above 

on-screen features? 

Nearly all participants were unable to determine that touch controlled 
the tested features. After exposures to touch (without any explanation) ard 
a short experimentation period, participants were asked what they thought 
controlled the tested features. Most participants thought mouse movement 
caused the feature effects. A list of responses are as follows: 

• Mouse movement 

• Mouse movement toward gray, shaded area (fading 
toolbars) 

• Proximity of hand to mouse 

• Heat of hand 

• The rear, red light on Sedona 

• Touch of the buttons on the mouse 

• I don't know 

After participants were told that touch of the mouse body controlled the 
tested features, all understood how to use touch and nearly all were intrigued 

by the novelty of an unfamiliar technology. 



S« verity 


Scope 


Issue 


Recommendation * 


2 


1.5 


Understanding how to accurately 
coiitral/use touch to operate tested 
features 


Use visual design cues to clearly identify 
that by touching the mouse will result in 
action. Participants mentioned using 
brighter colors for touch eteai, 
'squeezable' materials, and a touch- 
friendly product name. 



* Severity and Scope is explained in detail in Appendix C 



» In general, all participants were Impressed and Intrigued by the 
novelty of using touch to control features on the screen. 



3,0 Would users want to see these features In their next mouse 
purchase? 

• A majority of the users agreed that they would like to see these features 

in their next mouse they purchase. 

Feature YES 

Touch Tooltips 78% (7/9) 

Synar/Cursor on Touch ..67% (6/9) 

; Fading toolbars on Scroll : 67% (6/9) . 

Fading Tuuibdii* or Touch 56% (5/9) * 
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4.0 How do users rank these four features? 

• As a group, users did not establish a clear ranking for features. Feature 
rankings were not determined to be significantly different from each other 

(Friedman Analysis). *8 Subjects Re sponded 

Rating Frequency I 



Feature 


1 st 




3 ' 


mm 


Touch Tooltips 


3 


2 


2 


iillil 


Sonar/Cursor on Touch 


4 


1 


1 


lllll 


Fading Toolbars on 
Scroll 


1 


3 


2 


111! 


Fading Toolbars on 
Touch 


1 


2 


3 


iiiiii 



5.0 Which feature is most appealing and which feature is least 
appealing? 

More users felt that these features made the mouse more appealing 
than less. 

No user thought the touch tooltip feature made the mouse less appealing. 



Make the mouse more or less appealinq? 


fL 


Frequency 

5SS 1 , 12 3 4 5 6 7 


More] 


Feature 


Mean 


Less: More 
Appealing Ratio 


1 


2 


3 4 


5 


6 


7 


Touch Tooltips 


5.6 


0:8 




2 


- 2 

- 1 


2 


4 


2 


Fading Toolbars 


4.8 


2:7 




3 


3 


1 


Sonar 


4.7 


3:6 




2 


i l 


2 


2 


2 


Disappearing Cursor 


4.5 


1:6 


1 




- 3 


4 


2 





6.0 Do users find these features helpful or annoying? 

• Overall, more users felt that touch tooltips, disappearing cursor features 
were more helpful than annoying. 

Users were equally split in their feelings that the sonar and fading toolbars 
features were either helpful or annoying. 



Annoyinq or helpful? 


[»■<■ 


Frequer 


icy 

5 [Helpful] 


Feature 


Mean 


Annoying:Helpful 
Ratio 


1 


2 


3 


4 


5 


Touch Tooltips 


4.1 


1.9 




1 




6 


3 


Disappearing 
Cursor 


3.4 


2:5 


1 


1 


3 


3 


2 


Sonar 


3.2 


4:4 


2 


2 


2 




4 


Fading Toolbars 


3.0 


4:4 


2 




2 


2 


2 
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7.0 Do these features fit in with their preferred work style? 

. Overall, more users felt that all of these features fit into their work style than 

felt they didn't fit. 
• The touch tooltip feature fit the most users' work style. 



Fit users' work style? 




Frequer 

[Annoying] 12 31 


icy 

5 [He 


pftill 


Feature 


Mean 


Doesn't Fit: Fit Ratio 


1 


2 


3 


4 


5 


Touch Tooltips 


3.5 


1:6 


1 




3 


5 


1 


Sonar 


3.3 


4:5 


2 


2 


1 


1 


4 


Disappearing 
Cursor 


3.2 


4:5 


I 


3 


1 


3 


2 


Fading Toolbars 


3.0 


3:4 






3 


2 


2 



8.0 Was there too much going on in the screen? 

• More users felt that there was too much going on in the screen because 
of the fading toolbars and the sonar features. 



Too much qoinq on in the screen? 




Frequency 






[Ann 


ying] 


12 3 4 5 [Helpful] 


Feature 


Mean 


Disagree: Agree (that 
too much was going 
orORatio 


1 


2 


3 


4 


5 


Touch Tooltips 


2.3 


7:1 


1 


6 


2 


1 




Fading Toolbars 


3.3 


3:5 






2 


3 


?. 


Sonar 


3.3 


3:6 


2 


1 


1 


4 


2 



9.0 Do users find the mouse more or less touchable? 

. Overall, more users felt that addling touch capability to the mouse made 
it less touchable for all features. 



More or less touchable 


? 






Frequency 

v>; 1 2 3 A 5 [More] 


Feature 


Mean 


Less: More 
Touchable 
Ratio 


1 


t \ 3 


4 


5 


Touch Tooltips 


2.4 


6:2 


2 


4 ■ 2 


~2~ 




Sonar 


2.5 


6:4 


4 


m - 


3 


1 


Fading Toolbars 


2.7 


6:4 




3 [ - 


2 


2 


Disappearing Cursor 


2.9 


5:4 






4 





In general, participants felt that they would hold the mouse different 
because of the touch sensitivity. Participants mentioned they may hold 
the mouse lighter, be more conscious of where their fingers were on the 
mouse, and not rest fingers on mouse buttons. Furthermore, some felt they 
would keep hand off the mouse more or be anxious of removing hand from 
the mouse in fear of unwanted on-screen actions {causing the toolbars to 
vanish). These implementations of touch could encourage a completely 
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opposite user experience than our recent, form-fitting mouse designs which 
are intended to make mice more touchable use more comfortable. 

10.0 Participant reactions to fading toolbars 

In general, the initial reaction to fading toolbars was of surprise to participants. It 
was clear that most did not expect the toolbars to appear with mouse interaction , 
However, at least six participants mentioned they liked the fading toolbar feature on 
scroll. The three major obstacles for this feature are as follows ; 

Reappearing toolbars over text. "It is hard enough for me to find stuff or 
the screen, when I do find it, don't hide it from me!" This is a fundamental 
problem caused by showing users text that will be covered by the reappearing 
toolbar as soon as they move the mouse after scrolling or touching the mouse 
after typing. This causes users to have foresight to position the text of 
interest outside of the area where the toolbars will reappear. The grayed area 
on the screen to denote where the toolbars would reappear did not solve this 
problem. Participants mentioned that this single issue would cause them to 
use this feature less or not at all. 
Limited target user group. 

o users with large monitors, users who do not need/want additional 
screen space for viewing documents may not be interested in this 
feature. One participant with a 21" monitor mentioned he had no use 
for this feature. 

o Users who prefer to see toolbars 100% of the time. At least two 
participants mentioned they like having toolbars on all the time. 
• Longevity of feature use. Some participants mentioned they would 'give 
fading toolbars a try' or that fading toolbars x would add another learning 
curve' to their computing experience. At least four participants mentioned the 
feature was distracting. These comments may indicate that users might look 
to turn off this feature after continued use even though the feature intrigues 
them after a short exposure. 



Severity 


Scope 


Issue 

Reappearing too! be r over text (see 


Recommendation 

fto obvious solution, yet. 


2 


2 


Fading toolbars are distracting 


Allow users more control over when and 
how toolbars appear/disappear or to turn 
feature off. 


2 


2 


Touch of mouse doesn't accurately 
represent when user will want toolbar 
to appear or disappear 


More user research to determine when 
users do want toolbars and when they do 
not. 


2 


2 


Users don't understand what exactly 
controls the fade in and out of 
toolbars 


Provide more design cues to help users 
understand the cause and effect of this 
feature. 


2 


3 


User doesn't went passive touch of 
the mouse to control fade of toolbars 


Consider other mechanisms, such as a 
mouse button, to provide user with an 
active means to trigger toolbars. 


3 


1 


Fading toolbars do not perform 
consistently in all software 
applications 


Ensure consistent performance between 
all software applications. 


3 


1 


The fading toolbar feature causes 
slowed system performance 


Avoid reduced system performance, 
excessive writing to hard drive, or jerky 
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Severity 


Scope 


Issue 


Recommendation 










4 


3 


User wants a faster or slower or 
immediate fade in or out 


Allow users more control over when and 
how tuulbari appear/disappear . 


4 


3 


User expects toolbars to fade in after 
scrolling with scrollbar 


Make the toolbars fade in when user 
moves cursor out of scrollbar area. 


4 


3 


User wants to have fading toolbars in 
one application but not another (i.e. 
no fading toolbars in Word but want in 
IE) 


Allow user to turn fading toolbars on and 
off for different applications. 



11.0 Participant reactions to touch tooltips 

Participants generally welcomed this feature even thgugh they had same difficulty 
controlling when the tooltips appeared for bath Sedona and Viper mouse designs. 
Most participants felt having touch tooltips made using programmable mouse buttons 
easier to use. 

. 9 out of 10 participants agreed that controlling touch tooltips was 
much easier with Sedona than Viper. 

Sedona 

Participants generally found the performance of touch tooltips with Sedona to 
be usable but had some difficulty controlling the appearance and 
disappearance of the tooltip. Participants generally appreciated the idea of 
touching a button to get information about the use of that button 

The three major obstacles to this feature are as follows: 

• Controlling touch tooltips. Participants complained of having difficulty 
getting the tooltip to appear when they wanted it and also rot having it 
appear because of accidentally touching a touch-sensitive button. Some found 
the buttons too sensitive far resting their fingers on because with every tiny 
movement of their finger caused the tooltip to appear. Some found that 
simply using the programmable buttons was too difficult and required them to 
re-position their entire hand just to activate a side-button. 
Limited target user group. Participants tested were advanced or expert 
mouse users. In general, at least three participants were enthusiastic about 
having programmable buttons on a mouse. However, only a few participants 
mentioned that they would use the programmable buttons on Sedona. 
Longevity of feature use. Of those participants who would program the side 
buttons on Sedona, at least three mentioned they would not program the 
buttons to perform different tasks in different applications. They mentioned 
they would either not need a touch tooltip or would like to disable them after 
they become familiar with the button use. 



Severity/Scope issues for the Sedona mouse design: 



Severity 


Scope Issue 


Recommendation 


2 


l 


Inadvertent actuation and touch of 
side mouse buttons 


Consider the following; 
Include a time delay between touch of 
button and tooltip. Make mouse buttons 
less sensitive to touch (i.e. increase 
tooltip actuation pressure). 


2 


i 


Incorrect tooltip was displayed 


Ensure that correct tooltip is displayed 
with user touches front button and slides 



I !'o-;!,> cs= £-t:';e: I slecs.Tisiit v>t wwlfcui :-jt\jv£f text} 
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I fttt ^V.:.ff: lil f.^M.tfi'. ?i>ES. I ftfrt^ltii .Si'ttil'gS vl? *'.=0 S f! P '■.;'.:fl*-t!1j. Speed, 

j j and sensraiNiy to irwxHiriize user 

| i set&tecuor<. Aitovn users the abwrcy to 

j i turn feeture oft. 

Viper 

Nearly a\\ participants had dsr'hcuity using Vi par without aeiidentaily Usuchirci 
or actuating the right- or left-side buttons. Nearly alE participants rested their 
rirg fsngervrt ths rtght-skie button <srd tr-sit resulted »n e*c&&:v6, ifadv&rter-t 
toner toyhips dunno u.^e Pari id pants ai-Ks corornei-tso tnat pfTes=S:rg tr« 
S;3~- bi.tty caused them to «<£icte<*taily pr^ss sr^: rlght-skse tiytton. Hv.<U 
participants reported thny rsad to ne-pos^i&n their need in awkward position's 
to use the teft- <*rd f:ghr-s<Je butters, Participants indicated V:per was nearly 
unu»a&ie with active nghs- «nd lefu-stds; Buttons «rd os^omse ever more so 
wih thi addittor- of tout, h tooltips. Or* participant mentioned, "tr-at mow^* is 
anthiouch!" 



%y«Hfcy/S<:*jf** ,iss.M«s for the Vif?*r mofise <je$ign; 
Severity Scope Iccue 



Recommendation 



12,0 Participant reactions t® ^mappe^rmgf reappearing mowse 

The |fi"sp!«meniBitiyn of this ^at.ore was so twsp«3r«ni: Kj trifa participants that: r^ily 
a!! didn't rofsce that the cursor disappeared arid reappeared by tou'.h of the mouse. 
Only one participant mentioned i" :i s wo^d t<s hav« the <yrs&r vis:|jie -st asl tiroes;. 
Ha aiso rnentioeed that ha uses the L^e-iavior of his. cursor as an indicator o: his; 
sy stern's p^rfornuir-ce Witr-out it, he iii-s tiroes the system res igck&d. Ai= eth-^r 
participants welcomed the f&^iu-v. pr> parttcspdnt reported havinp tioobie f:rsd;n<| the 
turgor after resea^fe, ever- Vvh<eri tested without sonar. One particlpjs; t rrK-rftian^il he 
would i^e rhft following additions to ths cursor's- behavior. 

Would like cursor to center :ts;isif aftsr <; period of ron-ose 
* !Not oe at! is to go off th« screen 
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, Severity 


Scope Issue 


Recommendation 


2 


1 


Cursor relocates to the bottom right 
corner of screen at unexpected times 


c .;; >..M:t this bug 



13.0 Participant reactions to sonar 

Participants were quite taken back and surprised by the appearance of sonar. Some 
had mentioned they weren't sure what just happened, it was too profound, and that 
it bothers the eyes (like when you look at a camera flash). Some thought the idea of 
assisting the user in finding the cursor was great, but didn't appreciate the execution 
as presented, Participants were not clear on what controlled the appearance of 
sonar. Most thought it related to mouse movement. It was evident sonar assisted 
participants in finding their cursor, but not all participants wanted this assistance. 
The two major obstacles for this feature are as follows: 

• Limited target user group. Most participants had mentioned they don't 
need assistance finding their cursor on the screen. This feature may have 
been better received by users desiring assistance or as an accessibility 
feature. 

Longevity of feature use. Participants who reported they would use it all of 
the time also mentioned it may become more distracting over time This may 
be a feature that users look to turn off after a short period of use. 



Severity 


Scope 


Issue 


Recommendation 


2 


i 


Partial circles remain on screen or 
additional circles appear when sonar 
is near edge of screen 


Correct this bug 


2 


3 


User has a difficult time determining 
what controls sonar 


Use design to educate/indicate about 
touch, 


3 


2 


The circles are distracting 


Draw attention to the cursor more subtly. 
Consider lighter, smaller, and less circles. 
Also, instead of circles, consider making 
a different color cursor, brighter cursor, 
flashing cursor, or throbbing cursor on 


3 


2 


User wants to turn off or control 
speed of circles 


Allow users the ability to eustemi7e the 
performance of sonar 



14.0 Future research questions 

. How do users feel about touch? What is the best implementation? 

. What is the value of these features for users? Are these features compelling? 

Would users pay more for these features? Can we mimic 'touch' behavior with 

mouse movement? 

. When do users want toolbars present? Not present? How long would it be before 
users turn off the fading toolbars, if ever? What application and user groups does 
this feature compliment the best? 

. Will users program Sedona side buttons to perform differently in different 

applications suggesting they may need assistance remembering their functions? 
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How long would it be before users turn off the touch tooltip, if ever? What 
applications and user groups does this feature compliment the best? 
What is the optimal sensitivity of the programmable buttons to activate the touch 
tooltips? What is the optimal placement of the programmable buttons to control 
touch tooltips? 

Are there any long-term consequences of a disappearing cursor? Which 
applications support a vanishing cursor worst? How can we address these issues 
if present? 

How do we make touch more understandable with design? 
How do these features work together? 
Can touch enhance the gaming experience? 



15.0 On-line highlight media clips 



til\H[ E » v * v 3\Hsghlights\Tooltips.esf 

Reactions to fading .toolbars. 

\\timmuss-ril\HUE Web\HURproject\Touch-l\h!ghlights\Fading_Toolbars.asf 

Reactions to sonar 

\\tinimuss-ril\HUE Web\HURpro]ect\Touch-l\H!ghlights\Sonar.asf 
V>ti::i!>.uss-rir*UE V^;b^URpraject\Touch-l\hkihl:ght&\Toucri_vs._Movemerit.asf 
\\tsiirviiiss-rii\hUE web\HLkpt j]t:rt\fouch- l\Highlights\Mouse_Gnp.asf 
\\tlnimuss-rll\HUE Web\HURpro]ect\To. !!.!■■ l\hlghlights\Marketliig_lrislght.asf 
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APPENDIX A: PARTICIPANTS 



Number: 
Gender: 

Average Age: 

: Age Group (years) 
Frequency 

Handedness 

Mouse Handedness 

Computer Use: 

Days per week: 
Hours per day: 
i Hours per day 
! Frequency 



10 

7 Male/3 Female 



39.2 years (25-56 years) 
20-29 30-39 



1 Left/9 Right 
10 Right 

All reported they use a computer everyday 
3-4 5-& 7-8 



Self-Perceived Computer Experience: 

Mean Experience; 5.6 (Scale: l = beginner; 7=expert) 

ie: l=Strongly Disagree ;_7= Strongly Agree) 
Question Mean 
My friends think of my as a knowledgeable source of information for 6.0 
computer software. 

\ If I bought a new computer, I would be likely to purchase a new mouse 5.9 

; (other than the device that was sold with my new computer) that better 
; suits my needs. 

\ I know a lot about the different brands of computer mice. 4.6 
\ I would put a lot of time into selecting a replacement mouse. S O 



T, Muss 

Using Touch ors Mou&e; Phase I 

APPENDIX B; IMAGES OF SEDONA AND VI FIR MOUSE DESIGNS 




Top: (left to rig nt; Image uf keyboard and twy m*ce used. Left; Se<k;re rnuuse 
design. Right: Viper rtious* design 
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APPENDIX C: SEVERITY AND SCOPE EXPLAINED 



Severity indicates the seriousness of the problem: 

u Severity 1 problems prevent users from being able to complete a task. 
Participants give up after a few tries or they need a hint to continue. For 
example, users consistently select an incorrect dialog option and do not 
know what else to do. 

o Severity 2 problems create significant delay in completing the task or 
create significant frustration for the user. Participants continue to get lost 
or to use inefficient methods to accomplish a goal. For example, the lack 
of feedback to users confirming what they have just done causes them to 
do the task over to make sure they did it correctly. 

o Severity 3 problems have as a minor effect on usability, or are 

enhancements that can be made in the future. Forexample, an unusual 
term in a dialog causes users to hesitate for a moment before making the 
correct choice. 

o Severity 4 issues are not problems per se, but are issues that affect fit 
and firish, perceived quality, and/or overall satisfaction. 

Scope indicates the proportion of users affected by the problem: 

o Scope 1 problems will affect all users, This type of problem is likely to be 

encountered by any user using the system, 
c Scope 2 problems will affect about half of the users, Not everyone will 

have this problem, but a lot of users will, 
o Scope 3 problems will affect less than 1/3 of users. Some users will 

encounter this problem, and if it is of high severity it could have a large 

effect on perception of the product. 
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Patent Disclosure 

Title of Invention: Touch Sensitive Input Controls for User Feedback of Software Functions 
Date-BB— — 

Document Authors): Ken Hinckley, Steven Bathiche, and Peter Denuisicn (for the pop-up UI designs) 

1. Prior Disclosure 

Has there been any disclosure of this invention outside of Microsoft Corporation and its subsidiaries? 

No 



2. Introduction 

Input devices such as computer mice, keyboards, or game controllers often include many secondary inputs such 
as b uttons, wheels, levers, Triggers, sliders, or other mechanisms. The user typically has no interactive means of 
discovering how to use each control without reading the manual, or just trying out each button to see what it 
docs. Furthermore, even if the user learns the assignment of the buttons and other controls in one game or 
application, another game or application may assign the same buttons or controls to different features. The 
present invention addresses this problem by having an On- Screen Display (tool tips, help, or other information) 
appear (or even with auditory feedback or tactile feedback) based on contact or extreme proximity of the user's 
hand with touch-sensitive button(s) or other controls. The user can thus merely touch different buttons er other 
controls to explore their function assignments. 



3. Problem Addressed 

"What is the motivation of the invention? 

This invention helps the user to discover the function of input device controls. It also reminds users of the 
current functional assignment cf input device controls. All of this infonnation is provided based on user contact 
with touch-sensitive portions of the input device case. The user does not actually have to press a button and 
then guess at what the resulting action was. Just by resting on the control in question, informative help text or 
further options to customize the device behavior can be displayed, prior to actually activating the control. This 
helps to familiarize the user with the controls, as well as to prevent unintended activation of an incorrect control 
when the user is not sure of its current function assignment. 

What was done prior in the field? 

Typically the user must rely on printed materials, or alternatively a help file or screen which appears on the 
display when the user clicks a button or selects a menu option to request help. Sometimes the software will 
have a special mode where the user can press a button and see the function of that button highlighted on the 
screen. 



4. Description of the Invention 

By touching a button or a control on an input device the user requests the function of that control to be 
revealed. This can be communicated by 1 ) visual feedback either on the screen or on the actual button or by 2) 
acoustic or tactile feedback. The concept of touch buttons is extensible to almost any input device — mouse, 
keyboard, trackballs, game pads, joysticks, steering wheels, monitors, and headsets (Game Voice) — on any 
number of applications — productivity, games, and utility. 
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Hardware: A touch or proximity sensor is used to detect when the users hand(s) or finger(s) are in contact or 
extreme proximity to some portion of the input device. One or more portion(s) of the input device detect 
contact and can generate unique messages for the host computer based on which portion(s) of the input device 
the user is touching. Note that the user may be touching more than one centre! simultaneously. In some cases, 
feedback (e.g. audio feedback) may be provided directly by the input device and may or may not be shared with 
a host computer. 



Interactive Behavior. Touch-sensitive buttons or other controls provide a tool- tip when the user touches the 
control. We have implemented multiple possible policies for when a tool- tip or other information or feedback 
should appear, based on user contact with the control, time-outs, and other information such as mouse 
movement. The following figure shows three such options, discussed with reference to an embodiment of a 
touch-sensitive mouse button that causes a tool tip to appear on the screen 



Option I: Tile button tool tip is popped up (after a brief time-out) 
whenever the user touches the button, whether moving or not. The 
purpose of the time-out is to suppress tile button tool rip if tile user 
briefly clicks the button or accidentally brushes against the touch 
sensor (as shown on the right). 



ni 

Release | |_|_ 



Option IL The button to^i tin i« ™,t n«™v>, 
up unless tile user stops moving AND is 
touching ibe button. Time-out as before. 



Option III: Only show button tool tip if 
mouse is still AND you see a rising 
edge on the touch sensor. Time-out as 



Tool Tip 
Visible o 



Another aspect of our touch-sensitive input control technique is the response of the tool tip or other feedback to 
user activation of the button or other input control. After the user activates the control, the user may no longer 
want the feedback of the resulting action. For example, at the time the user elicits the button (button down 
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event), the button tool tip can disappear instantly (or fade out). The button tool rip can also reappear after the 
button up event occurs if the user is still touching the button, or it can stay hidden. The system may also display 
visual feedback to tie the button click to the information displayed in the tool tip before the tool tip disappears. 



Option I: After a button d<iwn event, 
the button tool tip will not reappear 



Option II: After abutlon down event, 
the button tool tip reappears (after 
the usual time-out) if the user keeps 
touching the button. 




Another issue is the handling of simultaneous contact with multiple controls on the input device. If the user 
touches a new control while still maintaining contact with a previously touched control, the on-screen display 
or other feedback for the new control she uld be provided If the user then releases the new control, the on- 
screen display or other feedback can be dismissed entirely, or the feedback for the previously touched control 
can be reinstated 



Visual Feedback For visual feedback on the screen in response to user contact with a touch-sensitive control, 
there are many possibilities. One simple possibility is to display a standard tool tip, such as those illustrated 
below: 




Typically, the cn-screen display can be placed near the current mouse cursor position (regardless of the input 
device that the user touches - for example, a keyboard tool tip could appear next to the mouse cursor). In 
particular we have found popping up the on-screen display to the right ef and above the current mouse cursor 
position to be useful, since this does not conflict with traditional tool tips which appear when the user dwells 
with the mouse over an icon (tool tip appears to the right of and below the mouse) in the Windows operating 
system. The tool tip can follow the cursor as the user mo ves the mouse, or it can remain stationary at the point 
where it initially appears. The latter technique is easier to implement and is more efficient, and seems well- 
accepted by most users. Other alternatives are to display the infomiation at the center of the screen, at the 
bottom of the screen, or above the system tray icons. 
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More elaborate on-screen displays are also possible. Here we illustrate several examples that show how 
programmable keyboard hot keys and function ("F") keys co uld make their function visible on a per application 
basis. If a user simply touches the key (without depressing it) the on-screen display will indicate what the key 
is and/or what it does - without actually performing the function. This examples shows on-screen display of a 
keyboard hot-key, as well as mouse-selectable options to customize or view settings of the keyboard hot-key. 



Q My CtKWputBr | 

Another more detailed version might look like the following This informs the user what the key is and what it 

: Q My Computer j 

i launches MfsCswpubsr : ^ 

f \ Reessqr. HotKey \ Svis-c-i | 



To take this idea even further, a feature we are considering for some future release is the ability to create text 
macros (the ability to insert blocks of text when the hotkey is depressed). When used in this context, we could 
display what the key is assigned to and the actual text macro. The on-screen display window would resize 
according to the amount of text assigned to the text macro. 



v T«?sl Macro 

Tha«£ you far.-iycur or tier . S%9««t vM 
b«' 12-24 hours: 



£««s^»HotK«y '; Settings 



Note that this feature is also very useful to replace LED's or other indicators on key boards and other devices. 
For example, users would merely have to touch the 1MTJM lock, Scroll lock, and Caps lock keys and the on- 
screen display would indicate the state of the key - on or off. This feature is useful for wireless products that 
may not wish to include LED's for power consumption reasons. 



Acoustic Feedback 

Audio or tactile feedhack can he used in comhinati-nn with or separately from visual feedhack on the display In 
some cases, vis ual feedback is not apj>ropriate, or acoustic feedback may be desired because of the currently 
running application, the input device that is currently being used, or user preference. 

For example, our new Came Voice product consists of a headset microphone combination attached to a puck. This 
puck allows you to selectively talk to multiple combinations of people and teams you are playing with. Each person 
is automatically assigned a code (1,2,3..) and a team (A,B,C. ..). A problem arises when the player has to remember 
which code corresponds to which player and team. When the user wants a reminder to whom a button is assigned to, 
the current implementation is to press and hold that button. The user then receives audio feedback of the name via 
their headset. Pressing and ho lding the b utton takes time for this form of feedback. The concept of feedback based 
on touching the button can be applied here instead. 

5. Related Writings or Products 

Any related writings? If so, please identify. 
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We published papers (Hinckley, Czerwinski et al. 1998; Hinckley and Sinclair 1999) and filed patents on 
the basic concepts of touch-sensing input devices approximately 2 years ago. The present invention of Touch 
Sensitive Input Controls for User Feedback of Software Functions may need to be filed as a new patent or 
extension of the previous filings. 

Our previous filings that relate to touch- sensitivity include the following: 

• Proximity Sensor in a Computer Input Device (96414.1) 

• Method of Interacting with a Computer Using a Proximity Sensor in a Computer Input Device 
(116279.1) 

• A Technique for Implementing an On-Demand Display Widget through Controlled Fading 
Initiated by User Contact with a Touch Sensitive Input Device (1 I662C.3) 

• A Technique for Implementing an On-Demand Tool Glass for Use in a Desktop User Interface 
(116620.2) 

• A Technique for Implementing a Two-Handed Desktop User Interface for a Computer (1 15205.1) 



6. Microsoft Products 

Is this invention intended for use in any Microsoft Products? 

This feature is being considered for future mouse, keyboard, and possibly gaming devices as well. 



Hinckley, EL, M. Czerwinski, et al. (1998). Interaction and Modeling Techniques for Desktop Two-Handed Input . 

Proceedings of the ACM UIST98 Symposium on User Interface Software and Technology, San Francisco, 
Calif, ACM, New York. 

Hinckley, K. and M. Sinclair (1999). Touch- Sensina Input Devices . ACM CHI'99 Conf. on Human Factors in 
Computing Systems. 
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